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INVOLVEMENT  AS  A  PREDICTOR  OF  PERFORxMANCE 
IN  I/S  PLANNING  AND  DESIGN 


Abstract 

The  critical  role  of  computer-based  information  systems  in  a  firm's  competitive 
strategy  increases  the  need  to  effectively  manage  I/S  planning  and  design  processes. 
This  research  examines  how  the  involvement  of  two  key  organizational  roles,  i.e., 
users  and  I/S  professionals,  affects  the  performance  of  I/S  planning  and  design  teams. 
Involvement  is  conceptualized  as  the  level  of  influence  of  a  role  for  a  planning  and 
design  task.  A  problem-solving  model  is  used  as  a  general  model  of  task.  Results 
indicate  that  a  three-component  problem-solving  model  (problem  formulation, 
decomposition/distribution  and  solution/synthesis)  provides  a  valid  basis  to 
discriminate  between  levels  of  influence  among  roles  for  actual  I/S  planning  and 
design  teams.  Analysis  of  the  relationship  between  the  measure  of  involvement  and 
the  performance  of  the  teams  supports  existing  theory  that  user  involvement 
increases  performance  of  design  teams.  An  exploratory  analysis  of  patterns  of 
influence  among  roles  across  problem-solving  components  (termed  involvement 
strategies)  indicates  that  interactive  strategies  where  designers  and  users  are  highly 
involved  in  every  aspect  of  problem  solving  correlate  with  high  performance. 
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1.  Introduction 

The  importance  of  computer-based  Information  Systems  (I/S)  to  the  competitive 
capability  of  the  firm  appears  to  be  increasing  (Bakos  and  Treacy  1986;  Huber,  MS 
1984;  Rockart  and  Scott  Morton  1984).  To  the  extent  that  this  is  true,  the  capability 
to  effectively  plan,  design  and  implement  such  systems  is  a  critical  task  facing  most 
organizations.  Not  surprisingly,  there  is  a  significant  emphas's  in  the  I/S  research 
literature  reflecting  a  range  of  perspectives  on  how  to  effectively  accomplish  this 
task  (Kottemann  and  Konsynski  1984,  Martin  1982,  Zachman  1986).  Central  to 
many  of  these  research  efforts  is  the  premise  that  increased  involvement  of  key 
organizational  roles  in  these  tasks  will  increase  the  likelihood  of  achieving  the 
proclaimed  benefits  of  an  I/S  investment. 

Such  claims  are  based  on  a  variety  of  theoretical  and  empirical  work.  For 
example,  Zachman  (1986)  draws  an  analogy  between  I/S  planning  and  design  and 
the  field  of  architecture  to  support  a  model  of  design  that  involves  five  distinct  roles. 
He  argues  that  successful  I/S  planning  and  design  processes  require  these  roles  to 
effectively  coordinate  their  activities.  From  a  different  perspective,  Mumford  (1981), 
Kling  (1977)  and  others  argue  for  incorporating  key  constituencies  in  the  planning 
and  design  process.  They  claim  that  involving  a  variety  of  roles  improves  the 
likelihood  of  successful  implementation  by  affecting  issues  such  as  perceived 
ownership,  commitment  and  the  ability  to  reflect  a  wide  range  of  beliefs  and  values. 

This  research  investigates  the  premise  that  increased  involvement  of  key  roles 
(specifically  the  user  and  designer  roles)  improves  the  performance  of  I/S  planning 
and  design  teams.  Section  2  examines  the  concept  of  involvement  and  discusses  the 
use  of  influence  of  a  role  for  a  task  as  one  means  to  conceptualize  and  operationalize 
involvement.  Section  3  empirically  evaluates  the  validity  of  using  a  problem-solving 
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model  of  task  for  measuring  the  influence  level  of  the  user  and  designer  roles. 
Section  4  then  examines  the  validity  of  using  this  perspective  of  involvement  for 
predicting  the  performance  of  an  I/S  planning  or  design  team.  The  patterns  of 
influence  across  components  of  a  problem-solving  model  are  examined  and  used  to 
characterize  alternative  involvement  strategies  and  predict  planning  and  design 
team  performance.  Finally,  Section  5  discusses  both  the  limitations  of  this  research 
and  the  implications  for  future  research. 


2.0  The  Concept  of  Involvement 

2.1  Overview 

A  fundamental  premise  underlying  many  approaches  to  the  I/S  planning  and 
design  process  is  the  need  to  involve  a  range  of  stakeholders.  In  this  context,  a 
stakeholder  is  an  individual  or  organizational  unit  that  will  be  affected  by  or  can 
affect  the  performance  of  the  planning  and  design  team  (Henderson  1987,  Mason  and 
Mitroff  1973).  Therefore,  planning  and  design  is  viewed  as  a  purposeful  activity 
intended  to  produce  a  product  or  artifact  that  will  affect  the  behavior  of  one  or  more 
stakeholders  (Churchman  1971,  Henderson  1987,  McGrath  1984).  Design  team 
performance  is  assessed  in  terms  of  its  members'  ability  to  execute  design  tasks  as 
well  as  the  impact  of  their  efforts  on  the  organization  and  their  stakeholders. 

The  manner  by  which  stakeholders  interact  in  the  planning  and  design  process  is 
often  discussed  in  general  terms  of  participation  or  involvement.  For  example,  the 
need  to  involve  users  in  the  I/S  planning  and  design  process  appears  to  be  a  premise 
held  by  most  I/S  designers  (Boland  1978,  De  Brabander  and  Thiers  1984,  Edstrom 
1977,  Ginzberg  1981,  King  and  Rodriguez  1981,  Markus  1983,  Zmud  1981). 


However,  the  way  in  which  the  concept  of  involvement  is  operationalized  and 
measured  is  less  clear.  For  example,  Ives  and  Olson  (1984)  review  a  wide  range  of 
empirical  research  on  the  impact  of  user  involvement  in  I/S  design.  They  find  that 
results  are  mixed  and  trace  this  confusion  to  an  inadequate  definition  of  the  concept 
of  involvement. 

This  work  proposes  that  an  adequate  definition  of  involvement  must  reflect  at 
least  two  dimensions:  (1)  the  definition  and  selection  of  role  representation  and  (2)  a 
model  of  task.  The  notion  of  role  representation  is  inherent  in  the  concept  of 
involvement.  That  is,  involvement  requires  that  the  I/S  planning  and  design  process 
interact  with  individuals  representing  various  values,  experiences  and  skills.  A  role 
definition  is  one  mechanism  to  generalize  process  models  from  one  setting  to 
another.  In  this  work,  the  notion  of  role  is  a  representation  of  the  perspectives  and 
skills  embodied  in  key  participants  of  the  design  process.  A  role  is  defined  as  the  set 
of  knowledge,  processes  and  implied  values  that  are  expected  from  a  given 
individuaUs)  filling  a  role.  In  this  sense  a  role  is  a  part  played  and  can  be  defined  by 
expectations,  capabilities  and  policies  (Van  Maanen  and  Schein  1979).  Similarly, 
Simon  (1976)  has  argued  that  a  role  is  a  representation  of  a  set  of  premises  held  by 
individual(s)  filling  that  role.  Such  a  view  highlights  the  importance  of  the  values 
carried  by  individuals  filling  roles. 

Henderson  (1987),  Nutt  (1975),  Zachman  (1986)  and  others  have  identified  a 
range  of  roles  that  could  be  studied  in  the  context  of  I/S  planning  and  design.  A 
common  model  used  to  study  I/S  planning  and  design  has  focused  on  two  key  roles: 
the  user  and  the  designer  (Beath  1986,  Churchman  1971,  Henderson  1987,  Ives  and 
Olson  1984).  This  research  will  also  focus  on  these  two  roles.  As  will  be  discussed  in 
Section  3,  the  managerial  role  is  also  included  as  an  informant  type  and  used  to 
develop  the  influence  measurement  model.  The  potential  for  explicitly  incorporating 
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the  managerial  role  as  an  element  of  an  involvement  strategy  is  discussed  as  a  key 
direction  for  future  research. 

The  term  "role  representation"  is  used  to  emphasize  that  an  actual  planning  and 
design  team  seldom  interacts  with  every  individual  that  could  fill  a  given  role.  For 
example,  the  user  role  is  represented  in  a  planning  or  design  process  by  those 
individuals  identified  as  users  and  involved  in  some  way  in  the  planning  or  design 
process.  The  manner  by  which  such  involvement  occurs  is  often  unspecified  (Ives 
and  Olson  1984).  At  a  general  level,  an  important  distinction  is  whether  or  not  the 
role  is  represented  as  a  formal  member  of  the  planning  or  design  team.  A  team  may 
well  reach  beyond  these  boundaries  to  interview  or  interact  with  non-team  members 
that  represent  a  given  role.  Gladstein  (1984)  has  argued  that  this  boundary  activity 
will  significantly  effect  the  performance  of  a  group.  In  this  research,  teams  that 
explicitly  represent  the  user  role  as  a  member  of  the  team  are  the  focus  of  study.  The 
extent  to  which  the  team  interacts  with  non-team  members  (i.e.,  manage  their 
boundaries)  is  not  formally  addressed,  although  such  activity  is  clearly  a  major  part 
of  the  team's  activities. 

The  second  dimension  of  involvement  is  a  general  model  of  task.  Involvement 
connotes  a  state  or  relationship  that  results  from  a  set  of  actions  or  behaviors.  For 
example,  Ives  and  Olson  define  user  involvement  as  ". . .  participation  in  the  systems 
development  process  by  representatives  of  the  target  user  group"  (1984,  p.  587). 
Boland  (IS 78)  explores  user  involvement  by  comparing  two  radically  different 
processes  of  interaction  between  a  system  designer  and  user.  He  differentiates  this 
process  in  terms  of  the  responsibility  of  each  role  across  the  phase  design  task  model. 
King  and  Cleland  (1975)  suggest  that  combining  role  and  task  models  is  an  effective 
basis  for  definition  of  planning  and  design  methods.  Their  use  of  techniques  such  as 
Linear  Responsibility  Charts  (King  and  Cleland  1975)  to  define  design  strategies  for 


Decision  Support  Systems  represents  an  approach  that  explicitly  partitions 
responsibility  of  a  variety  of  roles  across  a  task  model.  Similarly,  McGrath  (1984) 
uses  task  as  a  basic  dimension  in  developing  a  taxonomy  of  the  study  of  groups.  In 
essence,  these  researchers  and  others  (Ancona  and  Caldwell  1987,  Hackman  1968, 
Rowe  1987)  suggest  that  understanding  involvement  requires  knowledge  of  how  a 
set  of  roles  affects  each  other  across  a  model  of  task.  Less  clear  is  what  constitutes  an 
appropriate  model  oi  task  for  the  study  of  I/S  planning  and  design. 

Finally,  a  key  issue  in  the  development  of  a  model  of  involvement  centers  on  the 
specific  variables  used  to  operationalize  this  involvement  and  the  implications  of 
these  definitions  for  measurement.  The  variables  most  often  used  are  participation 
and  influence  (Edstrom  1977,  Ginzberg  1981,  Henderson  1987,  Ives  and  Olson  1984, 
King  and  Rodriguez  1981,  Locke  and  Schweiger  1979,  Robey  and  Farrow  1982).  The 
use  of  participation  is  grounded  in  an  extensive  history  of  participatory  decision 
making  and  problem  solving  (Ives  and  Olson  1984,  Locke  and  Schweiger  1979, 
Mumford  1981).  The  use  of  an  influence  variable  is  justified  based  on  both  a  notion 
of  participation  and  the  need  to  reflect  power  relationships  (Pfeffer  1981).  For 
example,  Robey  and  Farrow  (1982)  suggest  that  participation  by  itself  may  not  be 
sufficient  to  reflect  the  process  of  resolving  goal  conflicts  that  often  occurs  in  I/S 
planning  and  design. 

While  some  research  has  suggested  that  participation  and  influence  are  related 
to  performance  (Edstrom  1977,  Ives  and  Olson  1984,  King  and  Rodriguez  1981, 
Olson  and  Ives  1981,  Robey  and  Farrow  1982),  evidence  of  the  quality  of  the 
measures  used  in  studying  this  relationship  is  seldom  assessed.  And  yet,  the 
existence  of  method  bias  as  well  as  poor  reliability  in  the  measurement  of  influence 
for  a  buyer  decision  process  model  is  noted  by  Silk  and  Kalwani  (1982).  Similarly, 
Bagozzi  (1980),  Huber  and  Power  (1985)  and  others  have  pointed  out  the  sensitivity 
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to  method  in  obtaining  assessments  of  influence  for  complex  organizational 
processes.  However,  attempts  to  systematically  address  these  threats  to  validity  are 
lacking  in  I/S  design  research.  As  a  consequence,  it  is  not  clear  whether  the  mixed 
results  found  by  Ives  and  Olson  (1984)  are  due  to  poor  measures  or  inadequate 
theory.  This  issue  will  be  explored  in  depth  in  Section  3. 

This  research  will  operationalize  the  concept  of  involvement  as  the  level  of 
influence  exerted  by  a  role  using  a  problem-solving  model  of  task.  Specifically,  we 
will  examine  the  involvement  of  individuals  filling  an  I/S  planning  and  design  role 
and  those  filling  an  organizational  user  role.  A  managerial  role  is  reflected  in  the 
measurement  model  but  a  detailed  assessment  of  the  impact  of  this  role  on  the 
performance  of  the  team  is  beyond  the  scope  of  this  paper.  In  the  following  section, 
the  rationale  for  using  a  problem-solving  model  for  task  in  the  context  of  I/S 
planning  and  design  is  discussed  and  the  particular  problem-solving  model  used  in 
this  study  is  described. 

2.2  A  Measurement  Model  of  Involvement:  A  Problem-Solving  Perspective 

As  indicated  in  Section  2.1,  this  research  will  examine  how  the  involvement  of 
the  user  and  designer  roles  afTects  the  performance  of  an  I/S  planning  or  design 
team.  A  key  issue  is  the  model  of  task  used  to  operationalize  the  involvement 
concept.  One  approach  has  been  to  examine  the  involvement  across  a  life  cycle 
model  of  design.  Edstrom  (1977),  for  example,  assesses  the  level  of  influence  of  the 
user  at  difi'erent  points  in  the  systems  design  life  cycle.  His  findings  suggest  that 
increased  user  influence,  particularly  at  early  stages  in  the  life  cycle,  positively 
aflects  performance  of  a  team.  Similarly,  a  consistent  theme  in  research  on 
information  requirement  analysis  is  the  need  to  efiectively  involve  key  users  in  this 
stage  of  the  system  design  life  cycle  (Davis  1982,  Martin  1982). 
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The  research  presented  here  uses  a  problem-solving  model,  rather  than  the 
systems  design  life  cycle,  as  the  model  of  task.  Simon  (1981)  describes  the  design 
process  as  a  problem-solving  or  decision-making  process.  He  views  design  as  a 
general  task  domain  in  which  the  designer  must  generate  alternatives  and  choose 
the  "best"  design  solution.  This  view  of  design  provides  a  general  foundation  to 
apply  methodologies  and  concepts  from  m-anagement  ci.ience  to  the  pnhUms  of 
design.  It  emphasizes  the  search  and  choice  processes  found  in  design  and  provides  a 
linkage  between  the  research  on  design  and  the  research  on  human  problem  solving 
and  decision  making.  While  it  does  not  reflect  the  entire  range  of  issues  or 
perspectives  found  in  I/S  planning  and  design,  the  centrality  of  this  perspective  to 
the  I/S  field  justifies  its  selection  as  a  focal  point  for  research  on  I/S  planning  and 
design. 

A  variety  of  researchers  have  pursued  this  decision-making  perspective  of  design. 
Mostow  (1985),  Thomas  and  Carroll  (1984),  Adelson  and  Soloway  (1985),  Gross  and 
Fleisher  (1984)  and  others  have  proposed  decision-making  models  of  design  and  have 
conducted  empirical  studies  of  designers  to  support  this  perspective.  Empirical  and 
theoretical  work  ranges  from  micro- analysis  of  individual  designers  using  protocol 
and  observation  techniques  (Adelson  and  Soloway  1985,  Mostow  1985)  to  formal 
modeling  of  the  design  process  (Gross  and  Fleisher  1984,  Tong  1984).  While  these 
approaches  represent  a  wide  range  in  reference  disciplines  used  to  study  design,  they 
have  a  common  theme  of  design  as  a  problem-solving  or  decision-making  process. 

Often,  however,  the  decision-making  or  problem-solving  model  used  implicitly  or 
explicitly  to  study  IS  planning  and  design  is  individual-based.  Mostow  (1985),  for 
example,  uses  single  individuals  as  subjects  in  developing  a  descriptive  model  of 
design  behavior.  Similarly  Adelson  and  Soloway  (1985),  Thomas  and  Carroll  (1984) 
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and  Lawson  (1984)  also  base  their  empirical  work  on  observations  or  experiments  of 
individual  planners  or  designers.  While  these  perspectives  have  yielded  insight  into 
the  process  of  I/S  design,  such  models  have  two  significant  limitations:  (1)  lack  of 
communication-intensive  constructs  and  (2)  limited  emphasis  on  formulation 
processes  that  emphasize  cooperative  processes.  A  group  process  model  must 
explicitly  recognize  interaction  among  members  of  the  group.  McGrath  (1984)  and 
Witte  (1972),  for  example,  argue  that  communication  should  be  a  key  element  of 
models  used  to  study  group  behavior.  While  the  research  on  I/S  planning  and  design 
discussed  above  recognizes  the  relevance  of  communication  activities,  the  models  of 
problem  solving  do  not  readily  provide  constructs  to  explicitly  account  for  them. 

Secondly,  in  the  I/S  design  environment,  the  design  team  can  seldom  work  only 
collectively,  even  though  success  depends  upon  collective  action.  Rather,  the  team 
often  comes  together,  disperses  to  execute  tasks,  and  then  integrates  their  results. 
This  practice  coupled  with  the  centrality  of  decomposition  in  problem-solving 
behavior  of  designers  (Adelson  and  Soloway  1985,  Gane  and  Sarson  1979, 
Kottemann  and  Konsynski  1984,  Martin  1982)  suggests  the  need  to  highlight  how 
design  teams  decompose  problems  and  manage  the  assignment  of  work  among 
members  of  the  team.  Further,  the  opportunity  to  exploit  parallel  processing  in  the 
design  cycle  requires  attention  to  the  decomposition  stage.  In  such  circumstances,  a 
problem  must  be  decomposed  not  only  to  provide  conceptual  or  resource  bounds,  but 
to  maximize  the  ability  to  effectively  distribute  subproblems  to  dispersed  solvers. 

These  aspects  are  reflected  in  a  problem-solving  model  proposed  by  Davis  and 
Smith  (1983)  as  a  basis  to  study  the  interaction  between  multiple  agents  or 
knowledge  bases.  The  Davis  and  Smith  decision-making  model  is  defined  as  having 
five  components:  (1)  problem  formulation,  (2)  decomposition,  (3)  distribution,  (4) 
solution  and  (5)  synthesis.  The  problem  formulation  component  focuses  on  need 


identification  and  specification  of  relevant  environmental  contingencies.  Although 
Davis  and  Smith  do  not  explicitly  address  these  activities  in  their  research,  the 
importance  of  problem  formulation  in  planning  and  design  has  been  discussed  by 
Simon  (1981),  Churchman  (1971)  and  others.  Similarly  the  concept  of  a  generation 
activity  has  been  used  to  characterize  the  problem- solving  processes  of  groups 
(Hackman  1968.  McGrath  1984). 

Decomposition  focuses  on  dividing  the  large  problem  into  manageable  chunks.  It 
is  an  activity  recognized  as  a  critical  component  of  I/S  planning  and  design.  Studies 
of  individual  design  emphasize  the  use  of  decomposition  in  problem  solving  (Adelson 
and  Soloway  1985,  Carroll,  Thomas  and  Miller  1978,  Gross  and  Fleisher  1984, 
Lawson  1984,  Mostow  1985).  General  I/S  planning  and  design  methodologies 
emphasize  the  importance  of  the  decomposition  task.  To  the  extent  that  performance 
goals  encourage  the  use  of  parallel  work  strategies  among  team  members, 
decomposition  may  be  a  critical  task  for  the  I/S  planning  and  design  process. 

Distribution  focuses  on  work  assignment.  This  activity  is  the  focus  of  the  Davis 
and  Smith  (1983)  research.  They  investigate  the  knowledge  and  protocols  necessary 
to  effectively  match  a  problem  solver  with  a  problem  requirement.  As  suggested 
above,  distribution  is  quite  analogous  to  the  task  of  work  assignment  that  has  been 
studied  extensively  in  research  on  group  behavior  (Eils  and  John  1980,  Hackman 
1968).  This  activity  is  a  key  allocation  process  and  reflects  a  communication- 
intensive  aspect  of  group  problem  solving  that  is  often  left  implicit  in  individual- 
oriented  problem-solving  models. 

The  problem  solution  component  centers  on  the  activities  associated  with 
producing  a  solution  to  a  defined  subproblem.  The  Davis  and  Smith  (1983)  model  is  a 
distributed  problem-solving  model  in  that  it  is  possible,  if  not  assumed,  that  the 
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problem-solving  behavior  for  particular  subproblems  occurs  independently.  Of 
course,  such  a  requirement  is  not  necessary  if  the  team  works  as  though  it  is  a  single 
unit.  However,  the  settings  studied  in  this  research  involve  independent  work  on 
problem  solution  by  individuals  or  subgroups  within  the  team.  The  notion  of 
solution-centered  activities,  particularly  those  involving  choice  behaviors,  is  often 
highlighted  in  models  of  group  problem  solving  (Hackman  1968,  Laughlin  1980, 
McGrath  1984). 

Finally,  synthesis  focuses  on  the  need  to  combine  the  solutions  for  each 
subproblem.  Again,  this  activity  may  be  a  minor  issue  if  the  team  works  as  a  single 
unit.  However,  in  cases  where  subproblems  are  worked  on  independently,  the 
solution  process  may  in  fact  produce  conflicts.  In  such  a  case,  synthesis  becomes  a 
critical  task  for  a  design  team.  This  activity  is  emphasized  in  many  I/S  design 
methods.  For  example,  the  benefits  of  using  a  rigorous  syntax  in  requirements 
specification  are  often  justified  on  the  basis  of  an  improved  ability  to  combine  or 
synthesize  the  work  on  independent  design  processes  (Gane  and  Sarson  1979,  Martin 
1982). 

Figure  1  shows  the  five-component  problem-solving  model.  A  pilot  test  of  this 
problem-solving  model  with  project  members  from  six  design  teams'  indicated  that 
each  activity  level  is  a  distinct  component  of  work.  However,  the  highly  interactive 
nature  of  the  problem- solving  process  made  it  difficult  in  their  minds  to  specify  a 
single  sequence  of  these  activities  that  represented  the  team  work  process. 
Therefore,  the  model  does  not  imply  a  sequence  of  activities.  Rather,  the  problem- 


1  This  test  involved  members  of  each  team  filling  out  the  questionnaire  and  then 
discussing  each  item  in  depth.  Final  wording  tor  problem-solving  components 
resulted  from  these  pilot  test  interviews.  Interviews  lasted  approximately  one 
and  one/half  hours.  The  actual  definitions  used  are  shown  in  Appendix  A. 
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Figure  1 
Model  for  Role  Involvement 


Ir  =  Involvement  of  the  Rth  role 

PFiR  =  Level  of  Influence  for  the  Rth  role  in  Problem  Formulation 

DECOMiR  =  Level  of  Influence  for  the  Rth  role  in  Decomposition 

DISTiR  =  Level  of  Influence  for  the  Rth  role  in  Distribution 

PSiR  =  Level  of  Influence  for  the  Rth  role  in  Problem  Solving 

SYNiR  =  Level  of  Influence  for  the  Rth  role  in  Synthesis 


solving  model  is  represented  as  a  set  of  distinct  activities  or  task  components  for 
which  the  level  of  influence  of  a  given  role  can  be  reliably  attributed.  Of  course,  Silk 
and  Kalwani  (1982)  demonstrate  the  difficulty  of  developing  a  reliable  measure  that 
actually  discriminates  between  such  components.  In  the  following  section,  the 
validity  of  the  model  shown  in  Figure  1  is  tested  for  actual  planning  and  design 
teams. 
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3.0  Testing  the  Measurement  Model  for  Involvement 

3.1  Measurement  Approach 

In  this  section,  a  measurement  model  for  the  involvement  concept  is  tested. 
Involvement  is  defined  as  the  level  of  influence  exerted  by  a  role  for  a  job-related 
task.  In  this  case,  the  task  model  is  the  five-component  problem-solving  model 
discussed  above.  Bagozzi  and  Phillips  (1982)  argue  that  construct  validity  for  a 
measurement  model  requires  demonstrating  both  convergent  and  discriminant 
validity.  Evidence  of  convergent  validity  is  obtained  when  multiple  methods  for  a 
given  trait  correlate.  Evidence  of  discriminant  validity  is  obtained  when,  given 
convergent  validity,  a  test  of  the  difference  between  the  estimates  of  true  trait  value 
is  statistically  significant. 

Since  involvement  is  a  latent  variable,  i.e.,  it  cannot  be  measured  directly,  a 
Confirmatory  Factor  Analysis  (CFA)  approach  is  used  to  assess  the  measurement 
validity  for  a  distributed  problem-solving  process  model  (Bagozzi  1980,  Joreskog  and 
Sorbom  1979).  This  approach  has  been  used  to  test  a  variety  of  measurement  models 
in  disciplines  such  as  marketing  and  psychology  (Bagozzi  1980,  Bagozzi  and  Phillips 
1982,  Joreskog  and  Sorbom  1979).  In  particular,  the  CFA  can  be  used  to  examine  the 
convergent  and  discriminant  validity  in  a  manner  analogous  to,  though  more  precise 
than,  the  multimethod-multi trait  (MMMT)  approach  advocated  by  Campbell  and 
Fiske  (1959).  In  such  an  approach,  the  reliability  of  a  measure  can  be  assessed  by 
examining  the  correlation  among  multiple  measures  of  a  single  trait.  Evidence  of 
convergent  validity  is  provided  by  assessing  the  convergence  between  maximally 
different  measures  of  a  single  trait.  Discriminant  validity  is  assessed  by  testing  to 
determine  if  the  correlation  between  any  two  traits  (assuming  their  measures  are 
valid)  is  different  than  1.0  (Joreskog  and  Sorbom  1979).  The  use  of  structural 
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equation  models  permits  the  researcher  to  explicitly  test  for  both  convergent  and 
discriminant  validity  while  explicitly  controlling  for  potential  method  bias.  This 
approach  has  been  recommended  as  superior  to  a  traditional  MMMT  analysis 
because  it  provides  for  explicit  statistical  testing  of  the  discriminant  validity  for  any 
given  trait.  The  reader  is  referred  to  Bagozzi  (1980),  Bagozzi  and  Phillips  (1982), 
Joreskog  and  Sorbom  (1979)  and  Venkatraman  and  Ramanujam  (1987)  for  detailed 
discussion  of  this  approach. 

The  research  methodology  used  to  examine  the  involvement  of  roles  for  actual 
planning  and  design  teams  is  retrospective  key  informant.  Each  member  of  a  given 
team  is  asked  to  inform  on  the  level  of  influence  of  every  other  member  of  the  team  for 
each  component  of  the  problem-solving  model.  In  addition,  each  individual  provides  a 
self  report  (i.e.,  respondent)  of  his/her  own  level  of  influence  for  each  problem-solving 
activity.  A  t-test  to  examine  the  hypothesis  that  the  self  evaluations  differed 
significantly  from  the  evaluations  of  that  individual  by  informants  indicated  no 
significant  difference.  Thus,  there  is  no  evidence  of  a  systematic  bias  by  respondents 
and  this  data  is  included  in  subsequent  analysis. 

This  approach  results  in  multiple  informants  assessing  the  influence  level  of  each 
member  of  a  team.  A  structural  modeling  approach  can  be  used  to  test  for  the 
construct  validity  of  the  problem-solving  model  while  controlling  for  possible 
position  bias  reflected  by  a  given  informant  (Huber  and  Power  1985,  Silk  and 
Kalwani  1982).  This  influence  measure  is  operationalized  by  a  single  item  for  each 
component  of  the  problem-solving  model.  In  every  case,  the  influence  across  task  for 
each  individual  is  assessed  by  team  members  filling  three  roles:  manager,  designer 
and  user-.  In  general,  there  are  multiple  informants  within  a  role.  Assessments  of  a 


2  Role  assignments  were  based  on  descriptions  provided  by  both  the  project 
manager  and  the  individual  team  member.  In  the  few  cases  where  conflicting 
descriptions  occurred,  phone  interviews  were  used  to  clarify  the  individual's  role. 
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given  individual's  influence  from  informants  within  a  given  role  should  correlate, 
thereby  providing  a  means  to  assess  reliability  of  the  measure  of  influence.  As 
discussed  by  Silk  and  Kalwani  (1982),  to  the  extent  that  the  influence  assessment  for 
a  given  individual  across  informant  roles  also  correlates,  convergent  validity  is 
established.  Figure  2  shows  the  measurement  model  to  be  evaluated. 

Data  was  collected  for  18 1/S  design  teams  from  12  different  organizations.  In 
cases  where  several  teams  were  from  a  single  organization,  a  selection  process 
ensured  that  the  projects  were  significantly  different  and  affected  different  parts  of 
the  organization.  Thus,  each  team  represents  an  independent  data  point. 

In  general,  teams  included  7-12  people  from  which  5-7  informants  were  chosen. 
To  the  extent  possible,  a  random  selection  process  was  used  within  each  of  the  three 
major  roles'^.  All  measures  were  taken  within  three  months  of  the  completion  of  the 
project.  Thus,  assessments  reflect  the  judgment  of  informants  as  to  the  overall  level 
of  influence  for  an  individual  filling  a  role  during  a  project.  The  task  complexity  is 
controlled  through  both  the  size  of  the  project  (in  terms  of  dollars  and  personhours) 
and  the  total  length  of  time  between  project  initiation  and  completion.  The  time 
duration  of  each  project  was  approximately  9-12  months'*.  Because  of  the  limited 
duration  of  these  projects,  the  team  membership  was  quite  stable.  This,  of  course, 
suggests  that  these  teams  were  teams  brought  together  for  a  clear  purpose.  As  such, 
a  theoretical  model  that  views  design  as  a  problem-solving  process  is  appropriate. 


3  Exceptions  occurred  only  if  a  role  had  a  limited  number  of  individuals  filling  that 
role. 

4  Discussions  with  organizations  suggest  that  current  practice  defines  project 
goals  to  ensure  deliverables  within  this  time  frame.  In  many  cases,  the  team 
studied  is  part  of  a  larger  project  that  involved  many  deliverables  and  has  a 
longer  time  horizon.  In  fact,  the  product  of  one  team  is  often  a  starting  point  for  a 
subsequent  project. 
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Figure  2 
Five-Component  Problem-Solving  Model  with  Method  Factors 
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*    A's  within  a  method  are  defined  to  be  equal  to  reflect  a  common  method  bias  across  components. 
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Note:  Values  in  brackets  provide  five-trait,  two-method  solution. 
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Analysis  of  results  suggests  no  systematic  variance  for  years  of  experience  or 
assessments  of  level  of  expertise  within  roles  across  teams.  In  total,  there  were  126 
informants  participating  in  this  study. 

The  objective  is  to  assess  the  validity  of  using  the  problem-solving  model  to 
distinguish  levels  of  influence  of  roles  for  I/S  planning  and  design.  The  data  reflects 
teams  working  at  every  stage  over  a  limited  range  of  the  design  life  cycle:  planning 
and  design.  Projects  did  not  include  organizational  implementation  or  maintenance. 
Thus,  the  sample  does  not  reflect  a  full  range  of  product  development  as  discussed  by 
Ancona  and  Caldwell  (1987).  No  attempt  is  made  to  control  for  the  planning  or 
design  method  used  by  the  team.  For  example,  some  teams  used  prototyping  while 
others  used  structured  design  techniques.  This  study  will  also  examine  whether  the 
observed  level  of  influence  for  roles  across  tasks  (assuming  valid  measures)  supports 
existing  theories  of  the  relationship  between  involvement  and  performance.  To  the 
extent  that  such  a  relationship  holds,  research  can  then  focus  on  the  cooperative 
behaviors  that  generate  preferred  levels  of  influence.  At  this  future  point,  issues  of 
methodology  used  by  the  team  to  enact  an  involvement  strategy  would  be  a  central 
focus. 

3.2  Assessing  Convergent  and  Discriminant  Validity 

Analysis  is  conducted  using  a  set  of  three  interrelated  models.  The  first  model 
provides  a  baseline  for  evaluation  of  specific  hypotheses.  In  the  baseline  model,  each 
question  is  treated  as  an  independent  factor.  That  is,  the  baseline  model  assumes 
that  there  is  no  general  concept  of  involvement  that  is  being  measured  by  sets  of 
questions.  As  such,  it  provides  a  mechanism  to  characterize  the  total  variance 
reflected  by  the  instrument  and  can  be  used  as  a  baseline  for  assessing  the  portion  of 
variance  explained  by  a  particular  hypothesized  model  (Bentler  and  Bonett  1980), 
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This  model  includes  15  questions  reflecting  the  five  components  of  the  distributed 
problem-solving  model  and  the  three  roles  of  informants.  Each  measure  is  an 
average  of  all  assessments  of  the  informants  of  a  given  role  type  (method)  for  a 
particular  individual  for  a  given  problem-solving  component  (trait).  The  adequacy  of 
a  particular  hypothesized  model  can  be  tested  as  the  difTerence  in  two  chi-squared 
values  for  the  hypothesized  and  this  baseline  model.  This  difTerence  is  also  chi- 
square  distributed  and  removes  the  sensitivity  to  sample  size  associated  with  a 
structural  modeling  approach  (Bagozzi  and  Phillips  1982,  Joreskog  and  Sorbom 
1979).  Further,  the  difTerence  in  chi-square  can  be  scaled  by  the  baseline  chi- 
squared  value  and  used  as  an  index  (A)  of  the  goodness  of  fit  exhibited  for  the 
hypothesized  model  (Bentler  and  Bonett  1980,  Venkatraman  and  Ramanujam  1987). 
As  an  additional  test,  the  incremental  variance  accounted  for  by  subsequent 
restrictions  to  alternative  hypothesized  models  can  be  evaluated  by  and  will  be 
reported  as  a  modified  index  (A). 

The  second  analysis  hypothesizes  a  multiple-trait,  multiple-measure  model. 
Given  the  limited  access  to  team  members,  expanding  the  questionnaire  to  include 
multiple  items  and,  therefore,  strengthen  the  reliability  of  the  measure  was  not 
feasible.  Rather,  in  this  model,  each  influence  level  is  modeled  as  having  three 
measures  reflecting  the  source  of  the  information,  i.e.,  informant  type.  The  Xi  and  8i 
are  maximum  likelihood  estimates  that  partition  variance  into  a  trait  term  and  an 
error  term.  The  correlation  between  the  two  traits  (ij)  is  simultaneously  estimated 
by  the  correlation  (J)y.  As  will  be  discussed,  discriminant  validity  is  determined  if 
these  correlations  are  significantly  less  than  1.0.  In  essence,  this  model  tests  the 
hypothesis  that  the  variance  found  among  the  fifteen  questions  can  be  explained  by  a 
five-factor  model.  This  is  an  intermediate  analysis  used  to  provide  a  basis  for 
calibrating  the  efTect  of  evaluating  method  bias. 
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The  third  hypothesized  model  tested  is  the  basis  for  our  analysis  and  is  shown  in 
Figure  2.  This  model  extends  the  five-factor  model  to  provide  direct  formulation  of  a 
multimethod-multi trait  analysis.  In  this  analysis,  the  role  of  the  informant  is  also 
recognized  as  a  method  reflecting  the  possibility  of  a  systematic  bias  due  to  position 
or  knowledge.  The  principle  of  multimethod  analysis  calls  for  maximally  difTerent 
methods.  Thus,  while  this  analysis  will  provide  evidence  of  convergent  validity, 
additional  power  could  be  achieved  through  the  use  of  alternative  measurement 
approaches  such  as  direct  observation.  Given  the  limited  access  for  this  study,  such 
alternatives  are  not  available. 

In  this  multimethod-multi  trait  model  (Figure  2),  an  additional  set  of  Xf's  is 
incorporated  to  partition  the  total  variance  into  a  trait  component,  a  method 
component  and  an  error  component.  The  X's  estimated  for  a  method  across  traits  are 
assumed  to  be  equal.  This  is  an  interpretation  of  a  method  bias  as  a  systemic 
influence  not  related  to  the  structure  of  the  model  used  to  measure  involvement.  As 
will  be  discussed  below,  this  assumption  can  be  relaxed  to  provide  further  insight  as 
to  the  quality  of  any  given  informant  type. 

3.3  Analysis  of  Results:  Convergent  Validity 

A  summary  of  the  confirmatory  factor  analysis  using  LISREL IV  analysis  of  the 
baseline  and  the  three  hypothesized  models  discussed  above  is  shown  in  Table  1.  The 
results  indicate  that  the  five-trait,  three-method  model  (Figure  2)  accounts  for  90% 
of  the  variance  reflected  by  the  baseline  model.  The  modified  index  of  goodness  of  fit 
(A)  indicates  a  practical  significance  for  the  multimethod-multi  trait  model.  That  is, 
these  results  suggest  that  a  significant  amount  of  systematic  variance  is  due  to  a 
method  (informant)  bias. 
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Table  1 


Five-Trait  Measurement  Model  Results 


Model 

X2 

AX2 

A 

A 

prob. 

Baseline 

1274 

- 

- 

— 

NA 

5  trait 

570 

704 

.55 

~ 

.00 

5  trait. 
3  method 

126 

1148 

.90 

.35 

.00 

5  trait. 
2  method* 

41 

780 

.95 

- 

.01 

3  trait, 
2  method 

24 

578 

.96 

~ 

.04 

*  Baseline  X2  is  recalculated. 

Although  the  five-trait,  three-method  model  provides  an  adequate  fit,  further 
analysis  suggests  that  the  user  informant  may  be  an  invalid  source  of  information. 
Bagozzi  and  Phillips  (1982)  have  argued  that  convergent  validity  for  a  trait  is 
established  when  the  portion  of  the  variance  explained  by  a  given  indicator  for  the 
trait  is  less  than  the  portion  explained  by  the  aggregation  across  all  indicators  for  a 
given  trait.  That  is,  the  multiple  indicators  ..onverge  to  provide  a  more  reliable 
estimate  of  the  true  trait  value  than  could  be  generated  by  any  single  indicator.  This 
provides  a  type  of  measure,  called  pc,  that  is  analogous  to  Cronbacha  (1951).  It  is  the 
portion  of  the  variance  accounted  for  by  the  A  values  scaled  by  the  total  variance 
estimate  for  a  given  trait. 


The  comparison  of  the  pc  values  and  the  individual  Aj  is  provided  in  Table  2a.  As 
shown  in  Table  2a,  the  indicators  fail  to  converge  at  the  aggregate  level  for  the  traits 
Distribution  and  Synthesis.  The  likely  source  for  a  failure  to  converge  is  found  by 
examining  the  estimated  parameter,  trait,  method  and  error.  This  analysis  (Table 
2b)  indicates  that  the  method  bias  for  the  user  informants  is  particularly  high  and 
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Table  2a 


Convergent  Validity  Results  for  Five-Trait,  Three-Method  Model 


Indicator 
Trait       Indicator     Reliability  * 


Pc 


** 


PF 

M 

.31 

.37 

D 

.36 

U 

.24 

PD 

M 

.46 

.48 

D 

.43 

U 

.25 

DIST 

M 

.62 

.56 

D 

.55 

U 

.29 

SOL 

M 

.47 

.50 

D 

.50 

U 

.38 

SYN 

M 

.52 

.48 

D 

.52 

U 

.16 

*    The  value  of  ^.^2  can  be  viewed  as  the  indicator 
reliability  (see  Werts,  Linn  and  Joreskog,  1974) 

**  If  indicator  reliability  >  pc,  convergent 
validity  test  fails. 


the  trait  reliability  quite  small.  A  subsequent  LISREL  FV  analysis  permitting  the 
A's  for  the  user  method  to  be  independently  estimated  for  each  trait  (note  the 
previously  discussed  assumption  that  method  variance  is  constant  across  traits.) 
shows  the  X  values  (shown  in  brackets  in  Table  2b)  for  the  Problem  Formulation 
stage  to  be  consistent  with  the  other  methods  and  significantly  higher  for  the 
remaining  four  traits.  However,  the  reliability  of  the  trait  is  still  quite  low. 

An  interpretation  of  this  may  be  found  in  the  relationship  between  a 
measurement  strategy  and  the  theory  being  tested.  In  this  case,  involvement  is  the 
concept  being  measured.  The  measure  of  involvement  calls  for  informants  to 
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Table  2b 


Partitioning  of  Variance  and  Measure  Reliability 
Five-Component  Problem-Solving  Model  with  Method  Factors 


Component 

Indicator 

Variance 

Trait* 

Method 

Random 
Error 

PF 

Ml 

.30(.31) 

.28  (.25) 

.431.43) 

Di 

.36  (.36) 

.25  (.28) 

.41(.47) 

Ur 

.21  (.25) 

.49  (.27) 

.49  (.46) 

PD 

Mi 

.46  (.48) 

.28  (.25) 

.25  (.25) 

Di 

.43  (.41) 

.25  (.28) 

.28  (.28) 

Ui 

.2.5  (.23) 

.49  (.65) 

.17  (.12) 

DIST 

Mi 

.62  (.64) 

.281.25) 

.09  (.09) 

Di 

.56  (.53) 

.25  (.28) 

.15  (.15) 

Ut 

.29  (.27) 

.49  (.51) 

.23  (.24) 

SOL 

Mi 

.49  (.50) 

.28  (.25) 

.28  (.28) 

Di 

.51(.48) 

.25  (.28) 

.22  (.22) 

Ui 

.29  (.29) 

.49  (.41) 

.30  (.31) 

SYN 

Mi 

.53  (.55) 

.28  (.25) 

.19  (.19) 

Di 

.55  (.52) 

.25  (.28) 

.22  (.22) 

Ui 

.16(.17) 

.49  (.46) 

.36  (.38) 

This  can  also  be  interpreted  as  the  reliability  of  the  observed 
indicator  (Werts,  Linn  and  Joreskog,  1974) 


attribute  levels  of  influence  to  team  members  that  fill  a  given  role.  And  yet,  in  a 
majority  of  design  teams,  users  are  assessed  as  having  low  influence  in  the 
components  of  Problem  Decomposition,  Distribution,  Solution  and  Synthesis  (See 
Section  4),  i.e.,  low  involvement.  To  the  extent  that  lack  of  involvement  is  an 
indicator  of  isolation  from  key  team  activities,  an  informant  that  is  not  involved  in 
the  team  problem-solving  process  is  likely  to  be  an  invalid  source  of  information  on 
the  involvement  of  other  team  members.  In  fact,  these  data  suggest  that  the  user  is  a 
problematic  source  of  information  for  these  latter  phases  of  problem  solving^.  The 


5  Many  of  the  teams  include  non-I/S  professionals  in  the  manager  role.  Thus,  we 
conclude  that  the  lack  of  validity  is  not  due  to  inability  of  non-I/S  professionals  to 
interpret  the  question.  This  conclusion  was  also  supported  by  interviews. 
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poor  reliability  of  this  measure  coupled  with  the  high  method  bias  supports  the 
conclusion  to  eliminate  the  user  informant  as  a  source  of  measure.  Analysis  of  a  five- 
trait,  two-method  model  results  in  a  goodness  of  fit  of  95%  and  achieves  convergent 
validity  for  the  two  methods  used.  The  results  of  this  solution  are  shown  in  brackets 
in  Figure  2.  This  model  will  provide  the  basis  to  test  for  discriminant  validity  among 
traits. 

3.4  Discriminant  V^alidity 

A  LISREL  rV  analysis  can  now  be  done  to  determine  if  the  five-component 
problem-solving  model  demonstrates  discriminant  validity.  That  is,  do  informants, 
in  fact,  discriminate  between  the  five  components  when  assessing  the  influence  of  a 
team  member.  If  there  is  no  significant  discrimination  between  components,  the 
appropriate  action  would  be  to  average  the  five  influence  assessments,  treating 
involvement  as  having  a  single  dimension.  The  estimates  for  each  parameter  in  the 
adapted  five-trait,  two-method  model  are  shown  in  brackets  in  the  bottom  of  Figure 
2.  Analysis  of  these  results  indicates  that  the  least  differentiated  traits  are  Solution 
(trait  4)  and  Synthesis  (trait  5)  (4)45  =  .98).  That  is,  the  high  correlation  between 
solution  and  synthesis  suggests  that  the  informants  could  not  distinguish  between 
the  levels  of  influence  of  a  team  member  for  these  two  components  of  problem 
solving.  A  four-trait,  two-method  model  is  analyzed  to  test  the  hypothesis  that 
solution  and  synthesis  are  distinct  factors.  This  model  includes  the  necessary  and 
sufiicient  conditions  on  a  solution  that  hypothesizes  that  solution  and  synthesis  are 
equivalent  factors.  Specifically,  this  restriction  includes  defining 'J)45  =  1.0  and 
requiring  the  related  ^y  to  be  equal  (i.e.,  4)14  =  4>15.  4*24  =  "^25,  4>34  =  <J'35)-  The 
difference  in  X2  between  the  four-trait,  two-method  model  and  the  five-trait,  two- 
method  model  indicates  no  significant  difference  (X2  =  6,  d.f.  =  3,  p  s  .1). 
Furthermore,  the  goodness  of  fit  index  (A)  remains  high  (A  =  .93).  Given  this  result. 
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a  new  model  involving  four  traits,  two  methods  is  solved.  In  this  case,  the  responses 
for  solution  and  synthesis  are  averaged. 

Analysis  of  the  LISREL  IV  solution  for  this  four-trait,  two-method  model  also 
shows  a  high  correlation  between  the  problem  decomposition  and  distribution 
components  (4)23  =  .95).  A  similar  restricted  model  used  to  test  for  discriminant 
validity  of  these  two  factors  indicates  that  these  two  traits  can  be  modeled  as  a  single 
trait  without  significant  loss  in  explanatory  power^.  Subsequent  tests  for  difference 
among  the  remaining  three  traits,  i.e.,  (1)  Problem  Formulation,  (2)  Decomposition/ 
Distribution  and  (3)  Solution/Synthesis  support  the  conclusion  that  these  are 
distinct  traits.  Thus,  a  three-trait,  two-method  model  is  used  to  estimate  influence 
levels  for  study  of  the  impact  of  involvement  on  team  performance.  The  results  of 
this  analysis  are  shown  in  Figure  3^.  This  model  reflects  a  three-component 
problem-solving  model.  The  distribution/decomposition  component  is  a  distributed 
problem- solving  model  of  task.  Further,  the  ability  to  develop  valid  measures  of  how 
influence  is  distributed  within  a  design  team  offers  a  strong  foundation  to  study  how 
planning  and  design  teams  cooperate. 


( 

6  The  chi-square  test  of  the  hypothesis  $23  =  1-0  can  be  rejected  at  the  .10  level 
using  a  two-method  model  (user  evaluation  deleted).  Given  the  practical 
implication  of  a  .95  correlation,  we  conclude  that  informants  cannot  effectively 
discriminate  between  these  two  traits. 

7  A  solution  to  a  three-trait,  two-method  model  cannot  be  specified  since  the 
number  of  estimated  traits  plus  1  is  equal  to  the  total  number  of  indicators  in  this 
aggregated  model.  These  values  are  obtained  for  the  four-trait,  two-method 
model  that  incorporates  the  necessary  and  sufficient  condition  that  problem 
solution  and  synthesis  are  the  same  trait. 
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Figure  3 
Three-Trait,  Two-Method  Measurement  Model 


*    \*s  within  a  method  are  defined  to  be  equal  to  reflect  a  common  method  bias  across  components. 


ML 
Parameter  estimate 
Xl  =  .48 
\2  =  .84 
A3  =  .68 
^4  =  .76 
A5  =  .64 
Ae  =  -72 
X7  =  .61 
As    =    -46 


ML 

Parameter    estimate 
61     =    .37 
82     =    .10 
53     =     .14 

84  =    .18 

85  =    .21 

86  =    .20 


ML 
Parameter    estimate 
4>12  —    .37 
<t>l3  =    .45 
•*  $23  =    .88 


X2(df  =  14)  =  24    p<  .04 
A     =    .96 


'Note:  Testof  hypothesis  that  (t)23  =  1.0  is  rejected. 
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3.5  Summary  of  Results  for  the  Measure  of  Involvement 

The  results  of  this  analysis  suggest  that  a  three-component  problem-solving 
model  provides  for  reliable  and  valid  measures  of  influence  levels  of  individuals 
filling  key  roles  on  a  design  team.  The  data  indicate  that  informants  could 
discriminate  between  influence  for  three  types  of  problem-solving  activities: 
problem  formulation,  probieni  decumposition/distrioution  anu  soiutioxi/syuthesis. 
The  lack  of  discrimination  within  the  latter  two  components  could  be  due  to  poorly 
phrased  questions,  so  final  rejection  of  a  five-component  model  may  be  premature. 
Still,  the  three-stage  model  introduces  the  important  aspect  of  problem 
disaggregation  and  allocation  that  is  logically  part  of  a  group  problem-solving 
process.  This  may  be  particularly  important  in  the  study  of  design  teams  that 
attempt  to  incorporate  parallel  problem-solving  activities. 

The  results  also  highlight  the  measurement  difiiculty  associated  with  the  use  of 
influence  to  operationalize  the  involvement  construct.  The  reliability  of  measures 
used  to  operationalize  the  involvement  construct  is  generally  about  .50.   Further, 
clear  method  bias  exists  across  informant  types.  Such  circumstances  may  help  to 
explain  the  mixed  empirical  results  in  the  impact  of  involvement  found  by  Ives  and 
Olson  (1984).  These  results  suggest  a  need  to  carefully  articulate  a  theory  of 
involvement  as  well  as  to  account  for  how  the  nature  of  an  informant's  involvement 
may  bias  one's  measures. 

These  results  suggest  that  the  concept  of  involvement  can  be  opera tionalized  as  a 
measure  of  influence  of  roles  for  a  problem- solving  model  of  task.  The  next  step  is  to 
assess  the  predictive  validity  of  this  measure  in  order  to  provide  evidence  that  the 
measurement  model  does  reflect  the  concept  of  involvement.  Of  course,  as  Bagozzi 
(1980)  suggests,  the  strongest  evidence  of  validity  is  shown  when  this  measurement 
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model  produces  results  that  are  consistent  in  other  opera tionalizations  of  the 
involvement  concept.  The  assessment  of  nomological  validity  is,  however,  beyond 
the  scope  of  this  work. 


4.0  Involvement  Strategy  and  Team  Performance 

In  order  to  examine  the  relationship  between  the  involvement  of  key  roles  and 
performance,  a  two-stage  analysis  is  conducted.  First,  the  predicted  validity  is 
evaluated  at  the  general  level  of  the  relationship  between  user  involvement  and 
performance.  To  the  extent  that  the  measurement  model  conforms  to  expected 
theory,  i.e.,  user  involvement  increases  performance,  there  is  evidence  to  suggest 
that  the  measurement  model  does  reflect  the  concept  of  involvement. 

A  second  analysis  will  be  conducted,  assuming  predicted  validity  is  established, 
to  explore  the  pattern  of  influence  relationships  exhibited  across  design  teams. 
Cluster  analysis  will  be  performed  to  identify  potential  classes  of  influence  patterns. 
These  classes,  termed  involvement  strategies,  will  be  evaluated  using  independent 
performance  criteria  (i.e.,  variables  not  included  in  the  cluster  analysis)  and 
interpreted  based  on  existing  literature.  Implications  from  this  exploratory  analysis 
for  research  on  I/S  planning  and  design  will  be  discussed. 

4.1  Measures  of  Performance 

The  performance  of  the  design  teams  is  assessed  by  non-team  stakeholders  in 
terms  of  both  efficiency  and  effectiveness.  The  number  of  stakeholders  per  team 
ranges  from  three  to  nine  with  an  average  of  5.3.  In  every  case,  stakeholders  are 
senior  level  managers  familiar  with  the  project.  Stakeholders  include  at  least  two 
representatives  from  the  user  organization  and  one  senior  I/'S  manager. 
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Table  3 
Measures  for  Design  Team  Performance 

EfTiciency  items 

the  amount  of  work  the  team  produces 

the  efficiency  of  team  operations 

the  team's  adherence  to  budgets  and  schedules 

EfTectiveness  items 

the  quality  of  work  the  team  produces 

the  efTectiveness  of  the  team's  interactions  with 

people  outside  the  team 

Item  Scale:         12  3  4  5 

very  low     low      average     high   very  high 


Table  3  shows  the  measures  for  each  dimension.  The  measures  are  averaged  and 
used  to  assess  performance  in  terms  of  the  general  concepts  of  efficiency  and 
efTectiveness.  A  Cronbach  a  of  .90  and  .89  respectively  suggest  an  adequate  level  of 
reliability  for  each  of  the  two  aggregate  measures  used.  A  third  item  for 
efTectiveness,  the  level  of  innovation  of  the  team,  is  not  included  in  this  analysis 
since  the  Cronbach  a  calculation  indicated  a  lack  of  reliability.  A  more  detailed 
evaluation  of  the  validity  of  these  two  measures  using  the  structural  modeling 
approach  described  above  is  not  possible  due  to  the  limited  sample  size  (n  =  18). 
Therefore,  this  analysis  of  performance  must  be  considered  tentative  and  used 
primarily  to  motivate  future  research. 

An  informant  approach  to  performance  evaluation  is  used  because  on-site 
interviews  indicate  wide  variance  across  organizations  in  the  approach  and  quality 
of  more  objective  measures  such  as  dollars  or  hours  reported.  For  example, 
interviews  suggest  a  practice  in  some  organizations  of  using  slack  resources  to 
sponsor  individual  projects  as  one  means  to  reward  efficiency.  Thus,  the  accounting 
system  would  not  reflect  actual  effiiciency.  Similarly,  teams  facing  budget  overruns 
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often  do  not  report  or  under-report  overtime  hours.  While  the  performance  measures 
used  are  subjective,  the  use  of  multiple  informants  provides  adequate  reliability  for 
this  exploratory  research. 

4.2  Predictive  Validity 

The  underlying  logic  for  this  research  is  that  user  involvement  is  positively 
related  to  design  team  performance.  As  discussed  in  Section  2,  this  view  is  widely 
advocated  as  a  practice  and  has  some  empirical  evidence  to  support  it.  Therefore,  if 
the  measures  developed  herein  truly  reflect  this  underlying  construct,  this  measure 
of  involvement  should  predict  performance.  The  results  of  a  regression  model  of  the 
form 

Perfi  =  a  +  bUj  +  c 

where  Perfj  =  I'th  performance  measure  (efficiency  or  effectiveness) 
a  =  intercept 

b  =  regressioncoefficientof  user  involvement 
Uj  =  influence  level  for  the  user  role  average  over  the  three 
components  of  the  problem-solving  model 
c   =  error 

show  that  user  involvement  is  statistically  significant  for  efficiency  {R2  =  .24,  F  = 
5.5,  p  <  .03)  and  effectiveness  (R  =  .47,  F  =  15.2,  p  <  .001).  Thus,  ^he  null 
hypothesis  that  user  involvement  is  unrelated  to  performance  is  rejected. 

These  results  are  consistent  with  the  theory  suggesting  that  user  involvement 
leads  to  improved  performance,  thus  there  is  support  for  the  conclusion  that  the 
influence  measures  provide  a  valid  opera tionalization  of  the  concept  of  involvement. 
This  conclusion  is  tentative  given  the  limited  sample  size  and  the  use  of  a 
retrospective  informant  approach.  A  retrospective  analysis  permits  the  possibility 
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that  a  project's  success  could  influence  the  informant's  perception  of  the  influence 
held  by  members  of  the  team.  While  using  non-team  stakeholders  to  assess 
performance  and  collecting  data  soon  after  project  completion  may  help  to  moderate 
this  effect,  future  research  that  measures  influence  levels  during  the  design  process 
is  warranted.  Still,  the  strong  findings  for  actual  design  teams  is  encouraging.  In 
the  following  section,  the  general  involvement  measure  is  disaggregated  to  the  three 
problem-solving  components  ,  i.e.,  involvement  strategies,  to  provide  ^^  means  to 
analyze  the  pattern  of  influence  existing  within  planning  and  design  teams. 

4.3  Involvement  Strategies 

Given  support  for  the  ability  to  measure  involvement  at  a  general  level,  a  more 
detailed  analysis  of  the  pattern  of  influence  between  roles  across  task  components 
can  be  conducted.  This  pattern  of  influence  is  termed  an  involvement  strategy  to 
emphasize  the  shift  in  analysis  away  from  the  impact  of  a  general  concept  of 
involvement  to  a  more  detailed  examination  at  a  task  level.  This  approach  is 
consistent  with  prior  research  that  has  examined  the  relationship  among  roles  at  a 
task  level.  Edstrom  (1977),  for  example,  used  the  system  design  life  cycle  model  as  a 
model  of  task  to  examine  how  the  criticality  of  user  involvement  might  vary  over 
task.  Given  the  use  of  a  problem-solving  perspective  of  task,  the  research  by  Boland 
(1978)  that  views  design  as  an  inquiry  process  is  particularly  relevant.  Boland 
(1978)  studied  the  impact  of  processes  of  interaction  between  a  designer  and  user  on 
performance.  He  used  a  communication  protocol  to  enact  two  alternative 
involvement  strategies:  a  designer-driven  and  an  interactive.  The  designer-driven 
strategy,  called  traditional,  places  the  user  in  a  relatively  passive  role  in  the  design 
process.  The  user  is  a  source  of  information  concerning  need  and  responds  only  to 
recommendations  for  solutions.  The  designer  however,  plays  a  proactive  role  in  both 
formulation  and  solution. 
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The  interactive  strategy  is  described  by  Boland  (1978)  as  a  dance.  This  strategy 
requires  both  users  and  designers  to  independently  formulate  need  and  solutions  and 
then  explicitly  critique  each  other's  products.  Interaction  is  viewed  as  an 
opportunity  to  teach  each  other  and,  hopefully,  move  toward  a  final  solution  that 
improves  upon  the  original  interpretations.  Boland  (1978)  found  that  this  strategy 
leads  to  improved  design  products. 

An  exploratory  analysis  is  conducted  beginning  with  a  cluster  analysis  using  the 
influence  levels  of  both  designer  and  user  for  each  of  the  three  problem-solving 
components:  problem  formulation,  decomposition/distribution  and  solution 
/synthesis.  A  hierarchical  cluster  analysis  using  the  Ward  method  (Anderberg  1973) 
is  performed  with  these  six  variables.  The  Ward  method  sequentially  combines 
observations  in  a  manner  that  minimizes  wi thin-cluster  variance.  Examination  of 
the  clustering  process  showed  that  three  major  clusters  are  formed.  The  subsequent 
step  that  combined  two  of  the  remaining  clusters  resulted  in  a  major  shift  in  within- 
cluster  variance  and,  thus,  is  an  indication  to  retain  three  clusters.  The  average 
variable  values  for  each  cluster  plus  the  overall  average  is  shown  in  Table  4a.  The 
clusters  labeled  designer-driven,  user-driven  and  interactive,  have  a  total 
membership  of  7,  6  and  5  respectively. 

The  first  step  in  interpreting  these  three  clusters  is  to  establish  that  they  reflect 
distinct  involvement  strategies.  Results  are  shown  in  Table  4b  for  efficiency  and 
effectiveness  for  each  of  the  three  involvement  strategy  types.  Since  the  sample  size 
is  small,  a  non-parametric  test  for  statistical  difference  between  types  is  shown.  The 
Kruskal-Wallis  (1953)  sign  rank  test  is  used  to  test  the  null  hypothesis  that  there  is 
statistical  difference  between  strategy  types.  This  test  is  an  extension  of  the  Mann- 
Whitney  (1947)  sign  rank  test  for  analysis  involving  more  than  two  populations. 
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Table  4a 
Involvement  Scenarios 

Overall  Average  Influence  Interactive 

D       U 


D       U 


PF 


PD/Dist 


SoySyn 


3.5 

4.1 

3.3 

2.3 

3.8 

2.8 

N  =  18 


User-Driven 


D        U 


PF 


PD/Dist 


Sol/Syn 


3.5 

4.4 

2.8 

2.6 

3.7 

3.4 

N  =  5 


PF 


PD/Dist 


Sol/Syp 


3.8 

4.4 

3.7 

2.8 

4.0 

3.7 

N  =  4 


Designer-Driven 

D        U 

I 

PF 


PD/Dist 


Sol/Syn 


3.4 

3.8 

3.4 

1.8 

3.7 

2.1 

N  =  9 


Table  4b 
Performance  for  Involvement  Scenarios 


User-        Designer-     General         Pairwise 


Interactive      Driven        Driven 


Test 


Test 


3.98 

3.62 

3.16 

p  <  .025 

I*DD      p<.05 
UD»DDp<  .05 
I*UD     p<.10 

4.21 

3.84 

3.36 

p  <  .05 

I*DD      p^.05 
UD  *  DDp  <  .05 
I*UD     p<.10 

4 

5 

9 

Efficiency 


Effectiveness 


Number 
of  teams 


Item  Scale:  12  3  4  5 

very  low         low         average        high       very  high 


-30- 


The  test  is  analogous  to  a  one  way  analysis  of  variance  and  tests  the  null  hypothesis 
that  teams  across  different  strategies  are  actual  members  of  a  single  population.  As 
shown  in  Table  4a  statistical  significance  exists  for  both  efficiency  (p  <  .025)  and 
effectiveness  (p  <  .05)  indicating  that  the  strategy  differs  significantly  in  terms  of 
performance.  If  one  assumes  that  the  samples  are  independent,  pairwise  non- 
parametric  tests  of  the  strategies  using  the  Mann-Whitney  (1947)  sign  rank  tests 
indicate  that  both  interactive  and  user-driven  are  superior  to  designer-driven  (p  < 
.05)  and  that  interactive  has  slight  performance  advantage  (p  <  .10)  over  user- 
driven.  Although  one  must  be  cautious  in  interpreting  the  means,  the  direction  of 
difference  between  mean  responses  for  each  strategy  type  also  supports  the 
hypothesis  that  interactive  design  will  produce  high  performance.  Similarly,  a  user- 
driven  strategy  is  preferred  to  a  designer-driven  strategy  but  may  not  represent  an 
optimal  involvement  strategy. 

Examining  the  average  influence  levels  with  an  involvement  strategy  for  each 
problem-solving  component  reveals  interesting  patterns.  First,  the  influence  level  of 
users  in  problem  formulation  is  consistently  high.  To  the  extent  that  the 
professionals  represented  on  the  teams  accept  the  conventional  wisdom  that  user 
involvement  is  good,  such  a  finding  is  not  surprising^.  Similarly  the  influence  of 
designers  in  problem  solution  is  consistently  high.  Again,  given  the  normal 
expectations  for  the  designer  role,  such  a  finding  is  not  surprising. 

More  interesting  is  the  differences  for  user  influence  on  problem  solution  between 
designer-driven  and  interactive  (and  to  a  lesser  extent,  designer-driven  and  user- 
driven).  The  shift  in  the  role  of  the  user  is  significant  and  corresponds  to  a 


8     Interviews  with  project  managers  indicate  that  this  belief  is  supported. 
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significant  difference  in  performance  level.  While  the  sample  size  limits  the  ability 
to  test  for  causality  these  findings  suggest  that  mechanisms  for  increasing  the 
involvement  of  users  in  solution  generation  have  high  potential  for  performance 
impact. 

Less  clear  is  the  impact  of  the  designer  role  on  team  performance.  The  user- 
driven  strategy  does  suggest  a  decrease  in  the  involvement  of  designers  in  problem 
formulation.  The  reduction  in  performance  is  consistent  with  the  argumeiits  of 
Boland  (1978),  Churchman  (1971)  and  others  that  the  designer  role  in  problem 
formulation  is  important  since  the  role  may  challenge  the  user's  mental  model.  Such 
a  challenge  could  help  to  stimulate  debate  and  therefore,  lead  to  a  more  effective 
problem  formulation.  To  the  extent  that  this  argument  holds,  a  decrease  in  designer 
involvement  in  problem  formulation  should  decrease  the  performance  of  the  team. 
The  direction  of  the  performance  measures  supports  this  perspective  (user-driven 
performs  lower  than  interactive)  although  the  weak  statistical  significance  (p  <  .10) 
suggests  the  need  to  be  cautious. 

Finally,  the  lower  involvement  of  both  designers  and  users  in  decomposition 
/distribution  is  also  striking.  Debriefings  of  participants  suggest  (and  are  supported 
by  the  informant  data)  that  the  managerial  role  dominated  this  activity.  This  result 
also  suggests  an  opportunity  for  exploring  how  increasing  the  involvement  of  user 
and  designer  in  this  activity  may  impact  performance.  Certainly,  new  computer- 
aided  design  technology  offers  many  alternatives  in  this  area. 

These  results  are  consistent  with  the  results  found  by  Boland  (1978)  and  the 
arguments  of  Churchman  (1971).  Extending  Boland's  strategies  to  reflect  an 
involvement  strategy  dominated  by  users,  i.e.,  user-driven,  may  reflect  emerging 
trends  in  the  decentralization  of  I/S  and  the  rapid  diffusion  of  I/S  concepts  and 
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technology  throughout  the  organization.  It  should  also  be  noted  that  while  the 
designer-driven  teams  are  clearly  the  low  performing  teams  (as  compared  to  teams 
in  this  sample),  the  fact  that  the  applications  are  business-oriented  could  bias 
results.  That  is,  in  a  highly  technical  setting  such  as  operating  systems,  a  designer- 
driven  model  may  show  improved  performance.  The  benefits  of  adequately 
accounting  for  task  characteristics  as  well  as  understanding  how  these  involvement 
patterns  are  enacted  are  discussed  in  the  following  section. 

5.  Implications  and  Conclusions 

Several  implications  stem  from  this  research.  First,  the  need  to  more  precisely 
define  and  carefully  measure  a  notion  of  involvement  is  supported.  Further, 
although  the  small  sample  size  and  the  use  of  a  retrospective  approach  requires 
cautious  interpretation,  these  results  support  research  by  Boland  (1978)  and  others 
that  suggests  that  using  valid  measures  of  involvement  will  show  that  interactive 
design  teams  achieve  high  levels  of  performance.  That  is,  we  have  evidence 
indicating  that  user  involvement  contributes  to  high  performance. 

Perhaps  more  importantly,  these  results  suggest  that  a  problem-solving 
perspective  is  an  effective  basis  on  which  to  operationalize  the  concept  of 
involvement.  Results  indicate  that,  using  this  approach,  valid  measures  of  how 
influence  is  distributed  among  key  roles  can  be  obtained  for  actual  design  processes. 
However,  the  difiiculty  of  this  measurement  task  requires  careful  attention  to  be 
paid  to  the  possibility  that  method  bias  may  exist.  Given  this  ability  to  measure, 
involvement  can  be  used  as  an  intermediate  process  variable  to  study  how 
methodology  and  technology  of  planning  may  affect  the  performance  of  design 
teams.  That  is,  normative  theory  relating  method  to  involvement  can  be  developed. 
In  turn,  these  involvement  patterns  can  be  used  to  predict  performance. 
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At  an  exploratory  level,  these  results  indicate  low  levels  of  influence  for  the  user 
role  in  decomposition  and  distribution  (see  Table  4a).  There  is  some  indication  that 
the  interactive  teams  generate  more  involvement  in  this  aspect  of  problem-solving 
but  the  levels  do  not  reflect  the  active  involvement  seen  in  solution  generation  and 
problem  formulation.  Rather,  it  appears  that  the  managerial  role  dominates  these 
activities.  To  the  extent  that  the  designer  and  user  roles  add  value  in  problem 
formulation  and  solution,  one  can  speculate  that  their  involvement  in  decomposition 
and  distribution  may  also  improve  performance.  For  example,  involvement  of  users 
and  designers  in  decomposition  might  balance  the  tendency  for  a  given  problem 
decomposition  to  be  overly  biased  by  resource  implications  or  allocation  issues  that 
are  a  primary  concern  of  the  managerial  role. 

Two  other  important  directions  for  future  research  involve  boundary 
management  and  the  impact  of  technology  on  design  team  performance.  The 
boundary  management  issue  reflects  the  need  to  account  for  how  design  teams 
involve  their  non-team  stakeholders.  For  example,  do  high  performing  design  teams 
utilize  different  boundary  management  strategies,  e.g.,  gatekeeper  versus  network, 
for  different  types  of  boundary  tasks?  Should  such  strategies  vary  depending  upon 
the  characteristics  of  the  design  task?  Certainly,  research  by  Gladstein  (1984), 
Tushman  and  Nadler  (1979)  and  others  suggests  that  these  differences  may  affect 
performance. 

Secondly,  the  results  suggest  ways  that  computer-aided  design  technology  might 
have  an  impact  on  design  team  performance.  For  example,  the  recognition  of  a 
decomposition/distribution  stage  highlights  an  opportunity  for  effective  technology 
support.  Many  current  I/S  design  aids  provide  support  for  functional  decomposition. 
It  is  less  clear  that  these  tools  actually  have  an  impact  on  the  group  process  used  to 
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decompose  and  distribute  tasks  to  subteams  or  individuals.  This  framework  can  be 
used  to  explore  such  an  issue.  In  general,  the  premise  that  significant  benefits  will 
result  from  using  technology  to  alter  the  way  teams  work  together  (i.e.,  involvement) 
than  by  automating  specific  subtasks  can  be  systematically  studied  within  the 
structure  of  this  problem-solving  model. 


-35- 

APPENDIXA 
Definitions  of  Five  Components  of  Problem  Solving 


Problem 
Formulation: 


Problem 
Decomposition: 


Distribution: 


Solution: 


Synthesis: 


A  major  step  in  managing  a  project  is  to  identify,  formulate,  or 
define  the  problems  that  must  be  solved.  The  initial  problem 
identification  defines  the  goals  and  the  overall  scope  of  the  project. 
As  work  on  the  project  progresses,  this  definition  will  be  refined  to 
address  new  technical  or  managerial  problems  that  emerge. 


A  second  major  phase  of  problem  solving  is  to  break  up  the  project 
into  smaller  subprojects  or  tasks  that  can  be  solved  more  easily. 
Although  problem  decomposition  is  often  done  by  an  individual  in 
organizing  his/her  own  work,  we  include  that  activity  under  the 
Solution  phase.  The  questions  in  the  Problem  Decomposition  phase 
refer  to  breaking  down  the  overall  project  into  manageable  tasks. 

In  the  third  phase  of  problem  solving,  tasks  are  distributed,  i.e., 
matched  with  problem  solvers.  If  the  problem  is  particularly  large, 
tasks  may  be  initially  distributed  to  sub-groups,  and  then  to 
individuals.  Please  respond  to  these  questions  in  relation  to  the 
initial  distribution  of  tasks  at  the  project  level,  rather  than  to  the 
subsequent  levels  of  distribution  within  sub-groups. 

Solving  or  completing  the  subprojects  or  tasks  that  have  been 
distributed  is  a  fourth  phase  in  reaching  the  total  solution.  In  this 
phase,  individuals  or  small  groups  perform  the  work  that  addresses 
the  tasks  assigned  to  them.  We  call  this  the  task  solution  phase. 

In  this  phase,  the  individual  solutions  to  tasks  are  integrated  to 
form  the  overall  solution.  Synthesis  may  be  accomplished  in  one 
effort  near  the  end  of  the  project,  may  occur  intermittently,  or  may 
be  a  continuous  activity  throughout  the  lifetime  of  the  project. 


This  person  has influence  in  defining  the  problems  addressed  on  the  (project  name)  project. 


Person  t 


Very  Low 
1. 


Low 
2. 


Average 
3. 


High 
4. 


Very  High 
5. 


Unknown 
6, 
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